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[57] ABSTRACT 

A plurality of serial neighborhood transformation pipe- 
lines are provided for simultaneously operating on adja- 
cent segments of a partitioned image matrix. Tech- 
niques are disclosed for bi-directionally transferring 
pixel data on the edges of adjoining segments of the 
image matrix between adjacent processors in a manner 
which minimizes the number of interconnections there- 
between. In such manner a parallel pipeline image pro- 
cessing system can be implemented in integrated circuit 
form while keeping the number of pins for each stage in 
the pipeline to a minimum. 

10 Claims, 3 Drawing Figures 
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stage thus performs one transformation over the entire 
PARALLEL PIPELINE IMAGE PROCESSOR image in contrast with the parallel array processor tech- 
nique which requires one processing stage for each 
TECHNICAL FIELD pixel in the image. One of the tradeoffs, however, with- 

This invention relates to image processors. In particu- 5 the serial neighborhood processing approach is speed 
lar, it involves a high speed serial neighborhood trans- since each pixel and its neighbors must be sequentially 
formation processor that operates on separate segments presented to the neighborhood transformation logic 
of a partitioned image matrix. circuit one at a time. Other examples of serial neighbor- 

hood processors are disclosed in U.S. Pat No.' 3,805,035 
BACKGROUND ART 10 t0 Serra and in Kruse's article entitled "A Parallel Pic- 

Neighborhood image processors are a class of devices ture Processing Machine", IEEE Transactions on Corn- 
that operate upon a first array or matrix of pixels to puters, Vol. C-22, No. 12, Dec. 1973. 
generate a second transformation matrix in which each Sternberg, in his U.S. Pat. No. 4,174,514 (hereby 
pixel has a value depending upon its value in the origi- incorporated by reference) and Kxuse in his article rec- 
nal matrix and the values of its surrounding or neighbor- 15 ognize that the speed of serial neighborhood processors 
ing pixels in the original matrix. These neighborhood could be increased by partitioning the image matrix. In 
processors are extremely useful devices and find partic- Sternberg's approach the image matrix is partitioned so 
ular utility for pattern recognition, image enhancement, that contiguous segments of the image could be. pro^ 
area correlation, automatic 'inspection systems and cessed simultaneously by two or more adjacent serial 
other similar image processing functions. 20 neighborhood processors. Sternberg also realized that 

In general, two diametrically opposite approaches provision had to be made in this partitioning scheme to 
have been taken to construct these special purpose bi-directionally transfer pixel data between adjacent 
neighborhood processors. One form is known as a par- se rial neighborhood processors. This transfer becomes 
allel array processor in which there is a single comput- necessary when the neighboring pixels of the pixel 
ing element or module for each pixel. Parallel array 25 j,eing transformed in one processor is contained in the 
neighborhood processors of this type are disclosed in im ^ G ma trix segment supplied to the other processor.. 
U.S. Pat No. 3,106,698 to Unger and various papers ^ ; 

relating to the Illiac III pattern recognition computer. SUMMARY OF THE INVENTION 

. Generally, these parallel array processors comprise a x|, e pre sent invention is directed to improvements to 
matrix of identical processing modules, each module 30 ^ broa(J concepts disclosed in the aforementioned 
including a memory register for storing the value of a u s pat Nq 4^74^4 In particular, it is the object of 
single pixel and a neighborhood logic translator for ^ - mycnli6n to provide a method apparatus for 
computing the transformed value of that pixel as a func- transferring selected pixels between adjacent processors. . 
tion of the present value of the pixel and its neighbhor- ^ serial neighborhood processing system., 

mg pixel values. Parallel connections are necessary 35 J ^ u tQ minimize ^ number of 

between the translator for each module and all of the between adjacent processors . B y minimizing 

neighboring memory agisters. . the number of connections a partiUoned serial neighbor- 

The principal advantage of these parallel array pro- p roce ssing system can be more easily imple- 

cessors is speed: TTie transformauon of Ae entire ^ mSL^SS^^ form. Those skilled in the 

matrix is performed substantiaUy sim 40 ^5S a^prelSe that the number of pin connections is 
principle disadvantage of the parallel array ^processor PP implementing devices in . 

configuration is complexity since this construction re- ? 8M , , T r rtf Jv nrim9 rv fl Hvflnf«o« 

quires one processing module for each pixel in the ^teg^ted circuit form '^^^^T^t 
image matrix. In most practical applications the matrix of Sternberg's seria neighborhood 1"^" » 
size must be relatively large in otfer to achieve high 45 wc * sto ^? U ^^^^ 

resolution. For example, when the input matrix is gener- produced in integrated circuit form at mmimal cost If 
ated by a state of the aA television pick-up tube it may too many pin connections are required the cost advan- 
be d ijtized into a matrix of up to about 1,000 X 1,000 . tage over parallel array processors is recced How, 
pixelf Accordingly, a parallel array processor would ever; the present invention permits integrated c^cuit 
require one mimoYrelS fabrication of each stage in such a manner t^t the 

ules. The construction of such a parallel array processor stages can be configured in a partitioned serial neigh, 
baccordmelv very costly ■ 1 " borhood- processing system to achieve a great increase 

Atrial neighborhood processor represents, an alter- in speed at comparatively low increase in fabrication 
native approach to parallel array neighborhood prbces- cost , . . . „ 

sors. Examples of serial neighborhood processors are 55 The present invention 

disclosed in U.S. Pat. No. 4,167,728 to Sternberg, U.S. those pixels lymg on the edges of the partitioned image 
Pat No. 4,290,049 to Sternberg et at, and U.S. Pat, No. matrix to adjacent processors before the pixel data is 
178,312 to Sternberg et al. These patents and applica- stored in the nei^borhood window registers of each 
tion are hereby incorporated by reference. Disclosed stage for processing. This approach requires only one 
therein is a system which employs a chain or pipeline of 60 pixel transferring connection between adjacent proces- 
individually programmable serial neighborhood trans- sors in a partitioned serial neighborhood processing 
formation stages. Each stage is capable of generating system. Preferably, some means is provided for keeping 
the transformed value of one pixel within a single clock track of the pixel positions as they are initially fed .into 
pulse interval. The output of each serial neighborhood each stage. In the preferred embodiment, an address 
transformation stage occurs at the same rate as its input. 65 counter is used for this purpose. When the contents of 
This allows the output of one stage to be provided to the address counter indicates that a pixel being fed into 
the input of a subsequent stage which may perform a stage is also necessary for performing a neighborhood 
different neighborhood logic transformations. Each transformation in an adjacent stage, control logic is 



04/27/2004, EAST version: 1.4.1 



4,484,349 

3 4 

activated to effect the transfer of that pixel to the other tively, the matrix may be obtained directly from a scan- 
stage, ning device such as a TV camera. The original matrix is 
In the preferred embodiment the partitioned data output in raster scan line format to a demultiplexer 
matrix is fed into the stages in a line by line raster. By circuit 40 under the control of a master controller 42. 
appropriately delaying the lines being fed to adjacent S Demultiplexing circuit 40 operates to divide each raster 
stages, the simultaneous cross-wise transfer of pixel data scan line into three different segments and load the 
to a given stage can be avoided and a single bi-direci- divided segments into buffer storage devices 44, 46 and 
tonal bus (for multi-bit pixels) can be used to effect the 48. The primary purpose of buffer devices 4 4 48 is to 
transfer between adjacent stages. regulate the transfer of data between the high speed 

DESCRIPTION OF THE DRAWINGS 10 memor y 38 and the pipelines 12-16 which may operate 

DESCRIPTION Or 1 HE DRAWINOa at a somewhat i ower speed Xo ^ end buffer storag e 

Still other objects and advantages of the present in- devices 44 48 may comprise first-in-first-out (FIFO) 

vention will become apparent to one skilled in the art shift registers. Each shift register would be at least as 

upon reading the following specification and by refer- wide as the line length for the matrix segment to be 

ence to the drawings in which: 15 processed by its respective pipeline. In addition, the 

FIG. 1 is a block diagram of one embodiment of a buffer devices 44-48 should be able to contain at least 

parallel partitioned serial neighborhood image process- two continguous lines for the reasons that will become 

ing system in accordance with this invention; apparent later herein. 

FIG. 2 is a schematic diagram of one of the stages of By partitioning the original image matrix in such a 

the system shown in FIG. 1; and 20 manner, each of the pipelines 12, 14 and 16 can operate 

FIG. 3 is a timing chart illustratihjg the loading se- substantially simultaneously on their respective contig- 

quence of pixels into the parallel pipelines of the system. uous portions of the segmented matrix. Transformation 

nFSCP TPTTfW OF THF PUFFFRRED criteria ValU6S defmin 8 of transformation to be 

DESCRIPTION OF cmisd Qut [n ^ stage h loaded frQm a program stor . 

EMBODIMENT 25 a ge medium 50 into each stage of the pipeline. The pixel 
The present invention envisions the use of two or data is then loaded into the first stage of each pipeline 
more pipelines of serial neighborhood processing which performs its programmed transformation on its 
stages. In FIG. 1 image processing system 10 is shown respective matrix segment. The outputs of the first 
with three pipelines 12, 14 and 16. However, it should stages 18, 24, 30 of the pipelines 12, 14 and 16, respec- 
be understood that this should not be construed as being 30 tively, are coupled to the second stages 20, 26 and 31 
a limited example since the number of pipelines can be and so on through the last stages 22, 28 and 32 in the 
increased or decreased depending upon the speed factor pipelines. The output of the last stages can be coupled to 
desired. Similarly, the length or number of stages in a utilization device 52 for using the transformed matrix, 
each pipeline may vary depending upon the types of Utilization device may be, for example, a robot or other 
transformation algorithms that the system is designed to 35 device which is actuated by certain transformation out- 
process, puts. Alternatively, the outputs of the last stages can be 
Each of the stages 18-32 is substantially identical recycled through the pipelines for further processings 
from a hardware standpoint Each stage includes neigh- or to external memory. 

borhood transformation logic 34 which operates to ; As noted above, each stage operates on a subset or 

generate a transformed output value as a function of the 40 window of pixels. In this example, system 10 operates 

value of each pixel and the values of its neighboring on a 3X3 array, of pixels represented by the window 54. 

pixels in the image matrix being processed. All of the It can be apreciated that when pipeline 12 is operating 

neighborhoods in the image matrix are sequentially upon pixel 16 that its right-hand neighboring pixels 

extracted by an array, of neighborhood window regis- actually belong to the segment fed to the pipeline 14. In 

ters making up neighborhood extraction means 35 45 fact, this problem occurs for each edge pixel to be pro-. 

(FIG. 2). The central pixel in each neighborhood is cessed by each pipeline. The present invention is partic- 

transformed according to preprogrammed criteria, and ularly directed to a technique for assuring that each 

then sent to the input of the next stage. The method by pipeline has access to the necessary pixels to perform a 

which the neighborhoods are extracted for examination valid neighborhood transformation even though some 

and transformation may vary somewhat. The aforemen- 50 of the neighboring pixels belong to matrix segments fed 

tioned Sternberg patents and patent application illus- to adjacent pipelines. To this end, each stage includes 

trate alternative approaches, the approach disclosed in transfer control circuitry 56 for bidirectionally control- 

the patent application being generally followed in. the ling the transfer of edge pixels to and from each stage, 

description of the present invention. In this approach a Additionally, load control logic 57-59 is provided for 

memory 36 serves as a recirculating line buffer for tern- 55 controlling the timing of the pixel feed to each pipeline, 

porarily storing a given number of scan lines of pixel An important aspect of this invention is that only one 

data of a given length as well as program instructions pixel transferring connection is required to transfer 

defining the transformation criteria used by the neigh- pixel data between adjacent stages in the pipeline. In the 

borhood transformation logic 34. Pixel data for each, embodiment shown in FIG. 1 there is one pixel transfer- 
neighborhood is loaded from memory 36 into the array 60 ring connection 60 between the stages 18 and 24 and 

of window registers in the neighborhood extraction one connection 62 between stages 30 and 24. It will be 

means 35 where they are operated upon by transforma- appreciated that the left and right-hand pipelines 12 and 

tion logic 34 and then discarded. 16 need only receive pixels from the right and left edges 

In the present invention the image matrix to be pro- of the matrix segment supplied to the middle pipeline 14 
cessed is partitioned or divided into three contiguous 65 whereas the middle pipeline 14 needs to obtain edge 

portions for processing by pipelines 12, 14 and 16, re- pixels from both the left and right-hand segments sup- 

spectively. The original image matrix is stored in a high plied to pipelines 12 and 16. The number of lines for 

speed memory device 38 such as a disc or tape. Alterna- these connections (and hence, output pins in the IC 
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package for each stage) will depend upon the number of connection 60 to stage 24. The contents of address 
bits per pixel position. In other words, if each pixel is counter 86 in stage 24 will be at zero (or other predeter- 
represented by only one bit then only one line is re- mined count) since a new line of pixel data has not yet 
quired for each connection 60-70 whereas if each pixel been fed from its memory segment source. Accord- 
is represented by eight bits then eight lines will be re- 5 ingly, address counter 86 in stage 24 couples line 76- 
quired. Hence, the terra "pixel transfer connection" through multiplexer 72 so that pixel number 4 may be 
means the number of lines required to transfer all of the loaded into memory 36 of that stage. At the next time 
data associated with one pixel. step the address countersof stages 18 and 24 are incre- 

Turning now to FIG. 2, one embodiment of the trans- mented. At this time stage 18 needs to have pixel num- 
fer control circuitry 56 is shown in detail. A 3:1 multi- 10 ber 5 in order for it to perform valid neighborhood 
plexer 72 accepts pixel data from three sources. One transformations. Accordingly, stage 24 activates tri- 
source is over line 74 which is coupled to the normal state device 82 to send pixel number 5 over connection 
pixel data stream for the matrix segment supplied to that 60 to stage 18. Stage 18 causes its multiplexer 72 to 
stage. Another input is over line 76 which supplies pixel couple line 78 to latch 80 so that pixel number 5 can be 
data from the left-hand stage (here stage 18) and a third 15 loaded into its memory 36. It can be appreciated that 
input is over line 78 from the. right-hand stage (here stage 18 thus now contains all of the necessary pixels for 
stage 30). The output of multiplexer 72 is connected to that line. 

a latch 80 whose output is connected to the input of At time step 6 load control 57 is caused to delay 
memory 36. Hence, pixel data from any of the three further transfer of pixel data into stage'18 for that time 
input lines 74, 76 and 78 may be loaded into memory 36 20 step. As will become apparent to one skilled in the art,, 
via latch 80 for processing by the neighborhood trans- the outer most pipelines in this partitioned scheme will 
formation logic 34. only receive transferred pixel data from one adjacent 

Pixel data in latch 80 may also be transferred out of inner stage whereas the inner stages of pipeline 14 re- 
the stage to. either of. the adjacent left or right-hand quires pixel data from two of its outer adjacent stages, 
stages. The transfer of pixel data to the left or right is 25. Consequently, in order to prevent a conflict, i.e. the 
controlled by the states of tri-state buffers 82 and 84, need for simultaneous transfer of data between the 
respectively. In such manner pixel data can be simulta- stages on connections 60-70, the feeding of the pixel 
neously loaded into memory 36 and also transferred to data must be delayed for one time step each time, a new 
the left or right-hand stages. Means are provided for line is fed to the outer most pipelines. This delay func- 
controlling multiplexer 72 and tri-state devices 82 and 30 tion can be accomplished by a wide, variety of devices 
84 as a function of the position of the pixels in the image and may, for example, take the form of a slave micro- 
matrix fed to the stage. In other words, some means processor or simply a suitable delay device controlled 
must be provided for keeping track of the pixel position by master controller 42. 

each line of image data being fed into the state. To this At time step 7, a new line of pixel data is fed to stage 
end; an address counter 86 may be used. Each stage 35 18 whereas stage 24 continues to be loaded by pixels in 
includes control lines CIS and DIV, as well as a clock the first line. At time step 8, pixel number 8 is loaded 
input CLK from master controller 42. Each pixel is fed into the memory of stage 24 and also transferred to the 
into the stage during one clock time. Every time a new right-hand stage 30 over connection 62. Again, the 
line of pixel data is fed into the stage one of the control reason for this transfer is because the stages in pipeline 
lines, (here, line DIV) changes state to indicate the be- 40 16 need the values of those pixels lying on the right 
ginning of a new line. As each new pixel is received the hand segment fed to the middle pipeline 14 in order : to 
contents of the address counter 86 is incremented. In perform valid neighborhood transformations. At time 
such manner it is possible to keep track of the position of step 9, pixel number 9 is transferred to the left oyer 
the pixel being fed to or required to be fed to the stage. connection 62 into the stage 24 of the middle pipeline 

The operation of the present invention will become 45 14. 
apparent by way of the following simplified example. This staggered loading process continues until the 
Assume that the line length of the original image matrix entire partitioned image matrix segments have been fed 
is 1 2 pixels wide and that the matrix has been segmented into the pipelines 12, 14 and 16. Reference to FIG. 3 will 
into three segments of four pixels in length as repre- confirm the fact that bidirectional transfer described 
sented in FIG. 1. Reference to FIG. 3 also aids in under- 50 above may occur over a single connection withouj 
standing the operation of this invention. Each of the encountering a conflict. It can be appreciated that this 
vertical columns of boxes are labeled according to the technique has significant advantages over the particular 
pixel number being loaded into the pipeline and the example shown in the U.S. Pat No. 4, 174,5 14 Sternberg 
contents of the address counter 86 in the transfer con- patent. In the disclosed example therein there are six 
trol circuitry 56 of each stage. At time step 1 pipeline 12 55 separate connections to and from adjacent stages and, 
begins to receive a new line of pixel data (here pixel 1). thus, requires a six fold increase in pin connections in 
The address counter contents is thus represented in the comparison with the present invention. One of the 
triangle in the box as having the number 1. The next major conceptual differences is that the pixel transfer is 
two time steps sees the loading of . pixels 2 and 3 into accomplished before the pixel values are loaded into the 
stage 18. At time step four, the contents of address 60 neighborhood extraction means for accessing each 
counter 86 is at 4 and the latch 80 contains pixel number neighborhood for processing. According to the teach- 
4. Note that pixel 4 lies on the edge of the segment being ings of the present invention all of the pixel data neces- 
fed to pipeline 12 and is necessary for pipeline 14 to sary to perform each neighborhood transformation in 
perform a neighborhood transformation. Consequently, each pipeline is already present and loaded in the cor- 
- it is necessary to transfer pixel number 4 to pipeline 14. 65 rect sequence. Accordingly, there is no need to obtain 
This is accomplished by address counter 86 sending the edge pixel values from the storage devices making 
appropriate signals to enable tri-state device 84 such up the neighborhood extraction means in the adjacent 
that the value of pixel number 4 is transferred over processors. 
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Once the pixel data has been loaded into the first 
stage of the pipeline in the manner described above, 
each pipeline performs its programmed neighborhood 
transformation on its respective image segment, with 
the transformed pixel data progressing through each 
stage in the pipeline. This parallel partitioned approach 
serves to increase the processing speed of a single serial 
neighborhood processing pipeline at the expense of a 
relatively minor increase in cost. It should be under- 
stood that while this invention finds particular utility 
for processing image data, it is also useful for analyzing 
a wide variety of other types of data which is repre- 
sented as a matrix of values. Still other advantages and 
modifications of the present invention will become ap- 
parent to one skilled in the art after a study of the draw- 
ings, specification and the following claims. 

I claim: 

L A method of analyzing a matrix of pixel values in a 
serial neighborhood processing system, said system 
being adapted for sequentially performing a neighbor- 
hood transformation on pixel values contained in an 
array of neighborhood extraction storage devices, said 
method comprising: 

partitioning the matrix into a plurality of segments; 

feeding contiguous segments to adjacent neighbor- 
hood transformation stages in a raster scan line 
format; 

sensing the relative position of each pixel in the lines 
fed to each stage; 

bi-directionally transferring pixels lying on the edges 
of each segment to adjacent stages over a single 
connection means before said pixels are loaded into 
the neighborhood extraction devices for process- 
ing; and 

then, loading neighboring pixel values into said 
neighborhood extraction devices for processing 35 
whereby each stage contains all of the neighboring 
pixels for performing said neighborhood transfor- 
mations on the pixels fed to the stage even though 
some of. the neighboring pixels were initially fed to 
adjacent stages. 

2. The method of claim 1 wherein the feeding of each 
line of pixels to outer stages in the system is delayed by 
a suitable amount so as to prevent simultaneous transfer 
of pixels in a stage. 

3. In a parallel partitioned serial neighborhood pro- 45 
cessing system having a plurality of pipelines of serial 
neighborhood transformation stages, adjacent pipelines 
being adapted to perform neighborhood transforma- 
tions on contiguous segments of pixel data fed to each 
pipeline, wherein the improvement comprises: 

a single connection for pixel, transfer between adja- 
cent stages in the pipelines for bi-directionally 
transferring pixels lying on the edges of the seg- 
ments fed to adjacent stages. 

4. The improvement of claim 3 wherein each stage 
further comprises: 

transfer control means including multiplexer means 
for receiving pixel data from the matrix segment 
supplied to its pipeline and from said pixel transfer 
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and transfer of pixels to adjacent stages through 
said pixel transfer connections. 

5. A serial neighborhood transformation stage com- 
prising: 

neighborhood extraction means for sequentially ac- 
cessing a neighborhood of pixels in a matrix; 

transformation logic means for transforming each 
pixel in a matrix as a function of its value and sur- 
rounding pixel values contained in the neighbor- 

- hood extraction means; and 

transfer control means up stream of said neighbor- 
hood extraction means, operative to receive and 
transfer selected pixels between other stages. 

6. A parallel partitioned serial neighborhood process- 
ing system comprising: 

means for partitioning an image matrix into a plural- 
ity of contiguous segments; 

means for feeding contiguous segments to adjacent 
serial neighborhood transformation stages in a ras- 
ter scan line format; 

said serial neighborhood transformation stages in- 
cluding neighborhood extraction means for se- 
quentially accessing a central pixel and its sur- 
rounding pixels in the matrix, neighborhood trans- 
formation logic means for transforming the value 
of the central pixel as a function of its value and the 
values of its surrounding pixels contained in the 
neighborhood extraction means; 

a single connection for pixel transfer between adja- 
cent stages; 

means for sensing the relative position of each pixel in 
the lines fed to each stage; and 

transfer control means for bi-directionally transfer- 
ring pixels lying on the edges of each segment to 
adjacent stages over said pixel transfer connection 
before said pixels are loaded into the neighborhood 
extraction means for processing. 

7. The system of claim 6 wherein said pixel transfer 
connection is an eight bit data bus. 

8. The system of claim 6 wherein said transfer control 
means comprises: 

multiplexer means having a plurality of inputs, a first 
input being connected to said pixel transfer connec- 
tion from a left-hand adjacent stage, a second input 
connected to the pixel transfer connection between 
a right-hand stage, and a third input being con- 
nected to a supply of pixels from the image seg- 
ment fed to that stage; 

latch means connected to the output of said multi- 
plexer means, the output of said latch being cou- 
pled to said neighborhood extraction means, to said 
left-hand pixel transfer connection through a first 
gate, and to said right-hand pixel transfer connec- 
tion through a second gate; 

an address counter whose contents are a function of 
the relative position of each pixel in the lines fed to 
each stage; and 

wherein said address counter is utilized to control the 
multiplexer and first and second gates. 

9. The system of claim 8 wherein said feeding means 
is operative to delay the feeding of each line of pixels to 



connections from adjacent stages in other pipelines, 60 the outer stages by a suitable amount so as to prevent 



the output of said multiplexer means being coupled 
to neighborhood extraction means for temporarily 
storing a neighborhood of pixel values for process- 
ing and also being coupled to said pixel transfer 
connections; and means for sensing the relative 65 
position of the pixels in each segment line fed to the 
stage, operative to control the loading of the neigh- 
borhood extraction means through the multiplexer 



simultaneous transfer of pixels within a stage. 

10. The system of claim 8 wherein each stage includes 
a memory connected between the output of the latch 
and the neighborhood extraction means, operative to 
store a given number of scan lines and sequentially load 
said neighborhood extraction means with neighboring 
pixels. 

***** 
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